All committers have signed the CLA.

Thanks so much for the contribution so far! Looks great so far. Left some comments which may hopefully be helpful, and also just noting that loglikelihood_rolling is still todo.

Thanks for the comments @haileyschoelkopf I'm looking into implementing them today. I will also look into loglikelihood_rolling.

Update: I've implemented a function using loglikelihood_rolling with a test.

Will try to review this tomorrow! Looks great, thanks so much!

From a quick skim, I think there are just a couple minor things extant re: ensuring things break more than just logging an error message, if there's no/malformed response from the server. I can probably go and add those if need be.

A few-line pointer to add to the library README on what GGML apis are supported + a pointer on (where to go in order to) spin up an inference server for models would be awesome if not too much work!

Thanks @haileyschoelkopf! I updated the README with the command I'm using locally. I'm pretty new to evaluating models so I may be missing some details still.

Thanks! LLaMA-CPP is surprisingly easy to install... do you have any recommendations for models + tasks to test with GGML?

Running the 8bit LLaMA from here: https://huggingface.co/TheBloke/LLaMa-7B-GGML gives me 0.2504 accuracy on Hellaswag, which is I think way worse than I'd hope from an 8-bit quantized LLaMA-7b.

Thanks @haileyschoelkopf I'm getting similar results.

Update: I took out the reordering code which seemed to prevent the script from running. Pushed changes if anyone else wants to try.

Seems to be broken

I tried 85d3b8d and it fails to run

Running loglikelihood requests
  0%|                                                                                | 0/40168 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/idm/lm-evaluation-harness/main.py", line 93, in <module>
    main()
  File "/home/idm/lm-evaluation-harness/main.py", line 59, in main
    results = evaluator.simple_evaluate(
  File "/home/idm/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/home/idm/lm-evaluation-harness/lm_eval/evaluator.py", line 94, in simple_evaluate
    results = evaluate(
  File "/home/idm/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/home/idm/lm-evaluation-harness/lm_eval/evaluator.py", line 289, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/home/idm/lm-evaluation-harness/lm_eval/base.py", line 893, in fn
    rem_res = getattr(self.lm, attr)(remaining_reqs)
  File "/home/idm/lm-evaluation-harness/lm_eval/models/ggml.py", line 47, in loglikelihood
    response = self.ggml_completion(self.base_url, context=context, continuation=continuation)
TypeError: GGMLLM.ggml_completion() got multiple values for argument 'context'

I tried again with the previous commit 0fa2c7d and it runs but it returns a clearly incorrect result after doing a single completion (it finishes instantly instead of taking ~30min as you'd expect hellaswag, and returns the following results).

|  Task   |Version| Metric |Value |   |Stderr|
|---------|------:|--------|-----:|---|-----:|
|hellaswag|      0|acc     |0.2504|±  |0.0043|
|         |       |acc_norm|0.2409|±  |0.0043|

Thanks @IgnacioFDM I'll look into a regression there. The results I'm getting are even worse although I'm able to run it for longer iterations now.

just tried this out and getting the same issue as @IgnacioFDM.
Any luck getting this going @matthoffner ?

It seems llama.cpp is moving to a new GGUF format, so it might be good to change the name to GGUFLM

@matthoffner Can you post some evals run with the patched implementation, ideally with non-quantized #s for comparison?

@matthoffner Can you post some evals run with the patched implementation, ideally with non-quantized #s for comparison?

Will try to revisit this week, if anyone else can confirm that would be great too, thanks.

@matthoffner any luck?

@StellaAthena No luck. It runs without crashing but the results don't seem accurate.

@StellaAthena No luck. It runs without crashing but the results don't seem accurate.

Hmmm. That's unfortunate, though it could be that the quantization causes performance degradation.

Try manually inspecting a few of the results (esp. the incorrect ones) and then try running the same prompts through the library's native interface. Do you get the same logits when using the model through our library?

hey @matthoffner opened a PR against your fork: matthoffner#1 changing how the loglikelihood for completions is computed, following more closely what is done for the gpt3 and huggingface models.

For llama-2-7b-hf.gguf on winogrande I'm getting something close to what reported in the paper (there might some source of imprecision left). Before I was getting ~0.5 accuracy.

|   Task   |Version|Metric|Value |   |Stderr|
|----------|------:|------|-----:|---|-----:|
|winogrande|      0|acc   |0.6977|±  |0.0129|

Thanks @LorenzoMinto I merged your PR

This is the eval for arc-easy. Also slightly overestimated (in the paper they report 75.2), but quite close

{
  "results": {
    "arc_easy": {
      "acc": 0.7567340067340067,
      "acc_stderr": 0.008804009846865538,
      "acc_norm": 0.7373737373737373,
      "acc_norm_stderr": 0.009029861776763754
    }
  },
  "versions": {
    "arc_easy": 0
  },
  ...
}

These look good enough to merge to me, unless we want to confirm with other models and more datasets first.

@haileyschoelkopf Let's update the README to reflect this too!

Will do! Already done for the refactor branch PR

@haileyschoelkopf are you sure? It doesn't look like it.

In #967 (not yet merged) there’s the updated model support table with GGUF

In #967 (not yet merged) there’s the updated model support table with GGUF

Sorry, I shouldn't try to read things at midnight.